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ABSTRACT 


In this paper, a statistical model known as the AMSAA 
reliability growth model, is evaluated by measuring how 
well it tracks various reliability growth patterns. These 
patterns are defined by specified sequences of failure 
rates. Monte Carlo simulation is used to generate test data 
based on a time-terminated test plan. Estimates of failure 


rates, computed using the AMSAA model, are compared with the 


specified failure rates. Results for 18 reliability growth 


patterns are presented in both tabular and graphical form. 
Four modifications in the use of the model are discussed 


with selected results of their application presented. 
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T INTRODUCTION 


Many methods of modeling reliability growth have been 
proposed and used with some success. Reference 1 contains 
a detailed discussion of reliability growth, reliability 
growth models, and describes several models that are 
available to the manager for use in measuring reliability 
growth in his program. 

The purpose of this study is to evaluate the accuracy 
of the Army Materiel Systems Analysis Activity (AMSAA) 
reliability growth model [Ref. 1] for systems which provide 


limited test data during the development cycle. 


II. THE AMSAA RELIABILITY GROWTH MODEL 


The AMSAA reliability growth model is a statistical 
model that estimates reliability growth trends. It is 
applicable to systems for which usage data is measured on 
a continuous scale. 

As with most reliability growth models, the AMSAA 
model uses test data to estimate unknown parameters which 
are then used to estimate the reliability growth trend. 
References 1 and 2 develop the AMSAA model in detail. This 
paper addresses the use of the AMSAA model for sequential, 
time-terminated testing. 

Time-terminated testing occurs when each item is tested 
until failure or until that item has been tested a speci-~ 
fied amount of time. If an item does not fail before the 
planned test time, then a failure has not occurred and the 
next item is tested. 

For time-terminated testing, the AMSAA model [Ref. 1] 
provides the following estimate for the failure rate of a 
type of hardware unit (component, assembly, subsystem, 
system) at total accumulated test time T on that type of 


unity 
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where N = number of failures of that type of item to 
time T 
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and 


with X. = total accumulated test time over all items 
.tested when the j th failure occurs, 


Using these estimators, the accuracy of the AMSAA model 


was evaluated for a variety of reliability growth patterns. 


III. TESTING PROCEDURE 


Reliability growth models are often evaluated against 
actual test data. The problem with evaluating a model 
against actual data is that the failure rate driving the 
system at any one time is never known, it can only be 
estimated. 

The purpose of this chapter is to describe the test 
plan under which the analysis was made and define the test 
organization used. Figure 1 shows the relationship between 
the elements of the test plan and the organization of the 


testing. 


TEST PLAN 


PLANNED 
TEST TIME 


Figure 1. Test Organization 


A. TEST PLAN 

The key elements of the test plan used for the evalua- 
tion of the AMSAA model are described below, 

1. Phases 

Tests were conducted in phases consisting of testing 

a specified number of items with randomly generated exponen- 
tial failure times based on the failure rate specified for 
that phase. 


2. Failure Rates 


Each phase had a specified failure rate, which was 
assumed to be constant for the duration of the phase. The 
failure rates specified could be different from phase to 
phase. This set of failure rates defined the reliability 
growth pattern for a test, 

3. Items Tested 

A specified number of items was tested in each 
phase. In this analysis, the number of items tested was the 
same for each phase in any one test. 

4. Planned Test Time 
Each item in a phase was tested to failure or until 


the specified planned test time for that phase was reached. 


The planned test time was the same for all items tested 


during any one phase. 


B, TEST ORGANIZATION 
1. Cases 
The AMSAA model was applied to 18 test cases where 


each case was defined by a set of failure rates and a 


specified number of. phases, The general shape of the various 
cases, classified by number of phases and by trend of 
growth, can be found in Figures 2 through 5. Test phases 
are plotted, equally spaced, on the horizontal axis and 
failure rates are plotted sn the vertical axis. 
a, Set of Failure Rates 

Each set of failure rates defined a reliability 
growth curve which was then classified by its general trend 
of growth into one of two categories: 


(1) Non-increasing Failure Rates 
(Figures 2 and 3) 


(2) Failure Rates that Increase 
at least once during testing 
(Figures 4 and 5) 
b. Number of Phases 
In addition to their growth trend, cases were 
grouped, for comparison, by their total number of phases. 
The two groups are those that have 16 phases and those that 
have 6 phases. 
(1) 16-Phase Cases. Cases 1 through 6 
(Figure 2) and cases 13 through 18 (Figure 5S) were each 
defined by specifying 16 failure rates, one for each of the 
16 test phases, 
(2) 6-Phase Cases, The growth patterns for 
cases 7 through 12 (Figures 3 and 4) were defined by a set 
of six failure rates, Some of the cases reflect the same 


growth patterns shown for the 16-Phase cases, except that, 


the testing was carried out for only six test phases: 


Figure 2. 16~Phase Cases with Non-increasing 
Failure Rates 


a 


CASE 10 


Figure 3. 6-Phase Cases with Non-increasing 
; Failure Rates 


CASE 11 


CASE 12 


Pigure 4. 6-Phase Cases with Failure Rates 
that Increase 


CASE 13 CASE 14 


CASE 15 CASE 16 
CASE 17 CASE 18 


Figure 5. 16-Phase Cases with Failure Rates 
that Increase 


17 
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2. Data Sets 


Specification of the number of items to be tested in 
any phase for a particular case resulted in a Data Set. The 
number of items tested during a phase was the same for each 
phase of testing. Three sets of test data were generated 
for each case, one set each for tests with 5, 10, and 20 
items per phase. 

3. Tests 

The remaining element of the test plan, Planned Test 
Time, was used to define individual tests. Since the failure 
rate underlying the exponential test times was known, planned 
test times were chosen so that the probability of survival 
was such that sufficient failures would occur. 

a. Test l 

The planned test times for Test 1 were generated 
so that the probability of survival for each item would be 
85. 

b. Test 2 

Test 2 consisted of generating planned test times 
by using a probability of survival for each item of .939. 


This test was run to see how the model performed with very 


limited data. 


Iy. ANALYSIS PROCEDURE 


' A. MODEL ESTIMATES 
1. Shape Parameter 
Using the test data generated above, the shape 
parameter for the model was estimated at the end of each 


phase, i, using the AMSAA estimator: 


A N; 
oe ee 
N,inT; = £ I1nxX. 
jet 
where 
N; = number of failures that occurred to the 
end of phase i, 
T; = total accumulated test time over all 
items to the end of phase i, 
and X. = total accumulated test time over all 


J items when the j th failure occurs 
with a small exception. 
2. Failure Rate 
The model estimate of the failure rate at the end 
of the i th phase was computed in accordance with the AMSAA 


model as follows; 


Le a A a 


ltt should be noted that the estimate of 8 can only be 
computed when N,>1, If N,=1, the failure must not occur as 
the last item of a phase dince, i in that case, T,*X; » thus 
making 8 an indeterminate form. 


ge eS + + en ncaa enn eR See a no ge ge ex : 
ee — : eR FS ae 


This resulted in a set of failure rate estimates 


that was then compared to the set of specified failure rates 
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used to generate the test data. 


B. EVALUATION CRITERIA 

The manager using a reliability growth model is inter- 
ested in two things, accuracy and variability. 

1. Accuracy 

Accuracy measures the ability of the model to track 

a reliability growth trend. The measure of accuracy used in 
evaluating the AMSAA model was the failure rate estimate 
error as a percentage of the actual failure rate. 


This percentage error is defined as follows: 


[P5-74 

r.=-2. 

e, = ——*.x 100 , 
i Py 


x 
where r; = mean estimate of the failure rate for the 
i th phase 


and ry > specified failure rate for the i the phase. 


This measure of error was calculated at the end of 


each phase and used to trace the accuracy of the model over 


a Sample may be less than the 100 replications simulated. 
See footnote 1, page 17, 
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all phases of each test case. This estimate error, as a 
percentage of the actual failure rate, can be found in the 
tabulated statistics for each Data Set of the tests con- 
tained in the appendices. 

2. Variability 

A manager, considering a reliability growth model 
for use in a program, will not only be concerned with the 
model's accuracy, but will be concerned with the variability 
of the estimates the model provides. 

The simulation used for this evaluation was repli- 
cated 100 times. The mean estimates used to evaluate the 
tracking and measure the accuracy of the model only have 
value over the long run. A manager, however, is primarily 
interested in the one real application of the model. He is 
concerned with how well a model performs for a single 
replication and is thus ultimately concerned about the 
variability of the estimates he gets from the model in 
addition to its tracking accuracy. 

The measure of variability used for this study was 
the standard deviation about the mean estimate of the 


x 
failure rate, Pio 


ny & ene 
S.D.(r;) = Zz 2 (ps -r;3) 5 


where K = number of replications that yielded 
estimates. 
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As with the failure rate percent error, the standard 


deviation of the failure rate estimate is contained in the 


tabulated statistics for each data set. 


Vy. RESULTS 


A. GENERAL DESCRIPTION 

The results of the analysis are presented in the following 
manner. The output for each test is contained in a separate 
appendix to this paper. A data set is identified at the top 
of each page. This heading consists of the case number and 
the number of items for the data set. The output for each 


data set consists of two parts; the tabulated statistics and 


a performance plot. Figure 6 is a sample of the presentation 


of the results of a typical data set. 
1. Tabulated Statistics 
The input parameters, model estimates, measures of 
performance, and system status at the end of each phase of 
testing are tabulated for each data set. The values given 
are the average values over all replications for the data 
set. Figure 6 shows the format of the tabulated results. 
The following paragraphs describe each line of the format. 
a. Input Parameters 
(1) PHASE, This line indicates the phase of 
testing for the results listed below. Throughout the 
appendices, results for 6-and 16-phase growth patterns are 
presented. For l6ephase cases, the first 8 phases are 
tabulated above a row of asterisks, The same format is 


then repeated below the asterisks for phases 9 through 16. 
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Figure 6. Format for Results of a Data Set 
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(2) ACTUAL FAILURE RATE, This row consists of 
the sequence of specified failure rates that de*ines the 
reliability growth curve for the test case given in the 
heading. In Figure 6, the specified failure rates for 
Phases 8 and 9 are 0.1440 and 0.1330 respectively. 

(3) PLANNED TEST TIME. Here, the time to 
which each item was tested if it did not fail, is given. 
This number was based on the actual failure rate and the 
specified probability of survival for the test. This test 
time was the same for all items tested in a phase. For 
example, 10 items were tested in each of the 16 phases shown 
in Figure 6. Those items tested in Phase 8 were tested for 
1.1286 time units or until failure. 


b. Model Estimates 


(1) MODEL ESTIMATE. Line 4 of Figure 6 contains 
the estimates of the failure rate provided by the AMSAA 
model (see Section IV-A) for each phase of testing. Refer- 
ring to Figure 6, it can be seen that the model estimate of 
the failure rate at the end of the 8th Phase of testing was 
0.1681 as compared to the actual failure rate of 0.1440. 

ec. Measures of Performance 

(1) ESTIMATE ERROR, Here, the estimate error 
as a percentage of the actual failure rate, is given. Again, 
looking at the results for Phase 8, the percent error of 


16.75 is found. This was computed as follows; 
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0.1681 - 0.1440 


x 100 = 16.74 


0.1440 


The difference in the two percent errors is due to the extra 
accuracy carried by the computer. 

(2) SAMPLE STD DEVIATION. The next line of 
entries is the standard deviation of the medel estimate. 
This criteria is discussed in Section IV-B. The standard 
deviation of the model estimate for Phase 8 is 0.0685. 

d. System Status 

(1) CUMULATIVE TEST TIME, This row contains 
the total test time over all items as of the end of the 
indicated phase. At the end of the 8th Phase of testing, 
51.176 time units of testing had been performed on the 
system. 

(2) CUMULATIVE FAILURES. The final entry in 
the table is that of total failures on the system to the end 
of the phase. As illustrated by Figure 6, an average of 
12.01 failures occurred over all replications. This means 
that, of the 80 items tested to the end of the 8th Phase, 

12 failed. 
2. Performance Plot 
As an aid to eyaluating the tracking accuracy of the 
model, the specified failure rate and the estimated failure 


rate are plotted for each phase of a test, Figure 6 contains 


an example of a typical plot. 


All failure rates are plotted on a scale of 0 to 
1.2. If an estimate is greater than 1.2, it is plotted at 
1.2 and is noted as a point "out of range". More precise 
values of the points plotted can be obtained from the tabu- 
lated output above the plot. 

The phases, for each failure rate plotted, are 
scaled on the horizontal axis. By referring to the CUMULA- 
TIVE TEST TIME row of the tabulated statistics above the 
plot, the amount of test time on the system, to the end 
of a phase, can be found. 

For example, in Figure 6, the items tested had 
51.176 time units of testing at the end of Phase 8 and 
62.421 time units at the end of Phase 9. Thus test Phase 9 


consisted of 11.245 time units of testing. 


B. TEST 1 

The results for Test 1 are given in Appendix A. Model 
estimates, made when cumulative test times on the system 
were less than 10 hours, had a high percentage error and 
large standard deviations. Testing more items per phase 
tended to provide better estimates earlier in the testing. 

1, Non-increasing Failure Rates 

As would be expected, the AMSAA model tracks growth 

patterns that approximate continuously decreasing curves 
(Case 1) better than those that have drastic step changes 


(Case 6). The model had a tendency to provide estimates 


which lag behind the actual value of the failure rate. 


This is reflected in the increase in percentage error during 
the latter phases of testing. As can be seen in Case 4, 
several phases of constant failure rate will cause the 
model to lag behind even moderate changes in the failure 
rate. 

2. Failure Rates that Increase 

As can be seen in the performance plots of Cases 11 

through 18, the AMSAA model has a tendency to smooth 
oscillations in the failure rate. This also seems to be 
due to the tendency of the model to provide estimates of the 
failure rate that lag the actual failure rate when the 
growth pattern makes large step changes. This characteristic 
can cause some large percentage errors and standard devia- 
tions in the later phases of testing when the failure rate 


increases for any reason. 


C. TEST 2 

Test 2 used planned test times generated for a proba- 
bility of survival of .99. This resulted in far fewer 
failures during the testing which severely curtailed the 
model's ability to track a reliability growth pattern, 

Appendix B contains the results for some representative 
cases. As in Test 1, a minimum number of test hours or 
failures is required for the model to make an accurate 


estimate. This requirement is not met when a high relia- 


bility system is being tested. 


The results in Appendix B show a high percent error and 
a large standard deviation for the estimates made during 


the test. 


D. MODIFICATIONS 

As discussed above, the percentage error and the sample 
standard deviation of the model estimates were high during 
the early phases of testing and when failure rates in- 
creased. In an attempt to reduce this error, the AMSAA 
model was used in four modified test situations. All four 
tests used the point estimate of the failure rate 

number of failures during a phase 


r= ‘ 
P total test time during a phase 


It should be noted that the number of failures and the 
test time used to compute the point estimate are not 
accumulated over all phases but are only for the current 


phase of testing. 


The point estimate, 5 » was used for any phase where the 


cumulative test time on the system was less than 10 hours. 
Once the 10 hour threshold was exceeded, AMSAA model esti- 
mates were used, These estimates were made using all data 
from the beginning of testing if the failure rate was not 

considered to have increased. The four modified tests 


differed only in their handling of increasing failure rates. 


29 


PP ahh ene ee PaE ey eS Se ES ea 


1. Estimating Slope 


In order to determine whether the trend of failure 


rates was increasing, the slope of the trend was estimated. 
Two methods were used: 
a. 2=Point Estimate of Slope 
The slope was estimated using the point estimate, 
rp? for the current phase and the last estimate, Pit ; 


which may have been from the model or a point estimate. 


Thus SLOPE = nr - My) 
b, %3-Point Estimate of Slope 

This method of estimating slope was used to 
reduce the change that the estimated slope would be affected 
by a randomly high point estimate. The slope was estimated 
using the point estimate, rp , for the current phase and the 
last two estimates bios and os . The slope estimated con- 


sisted of two parts: 


For the estimate of the slope to be considered increasing, 
both parts had to be determined to be increasing. 
2. Determining Increasing Slope 
Two methods of determining when an estimate of 
Slope could be considered increasing were used. The esti- 


mates of slope described above were tested in one of the 
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following ways for each of the modified tests. 
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3. Modified Tests 
Figure 7 shows the composition of each of the four 


modified tests. 


SLOPE = .07 SLOPE 3 .2r;_ 
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Figure 7. Composition of the Modified Tests 


| 
| 
{ 


The results of the modified tests are presented in 


the appendices as follows: 


TEST APPENDIX _ 


MOD1 Cc 
MOD2 D 
MOD3 E 
MOD4 F 


Cases 4, .3, 15, and 18 are presented for each test, 


APPENDIX A 


Results of Test 1 


The results of Test 1 for Cases 1 through 18 are con- 
tained in this appendix. Each case consists of three 
data sets, one each for 5, 10, and 20 items. The results 
are presented in two parts; the tabulated statistics and a 
performance plot. The elements of the tabulated statistics 
are described below. For l6-phase cases, the first eight 
phases are presented above the asterisks and the last eight 
are shown below. 
1. Input Parameters 
a. PHASE 
This line indicates the phase of testing for which 
the elements listed below apply. All statistics are as of 
the end of the indicated phase. 
b. ACTUAL FAILURE RATE 
This line contains the failure rate specified for 
the indicated phase. This failure rate was used to 
generate the exponential failure times used as data for the 
model. 
ce. PLANNED TEST TIME 
This is the test time to which each item was tested 
if failure did not occur earlier, For Test 1, the planned 


test times were generated so that the probability of 


survival was ,85, 


2. Model Estimates 
a. MODEL ESTIMATE 
This is the estimate of the failure rate provided 
by the model for the test phase indicated (See Section IV-A). 
3. Measures of Performance 
a. ESTIMATE ERROR AS 
PERCENTAGE OF ACTUAL 
FAILURE RATE 
These entries were computed as a measure of accuracy 
to be used for evaluating the tracking accuracy of the AMSAA 
model (See Section VI-B). 
b. SAMPLE STD DEVIATION 
These values were computed as a measure of varia-- 
bility for the estimate error (See Section IV-B). 
4. System Status 
a, CUMULATIVE TEST TIME 
The times listed are the total test time accumulated 
over all phases of testing to the end of the indicated phase. 
b. CUMULATIVE FAILURES 
These entries are the total failuares over all items 


tested for all phases through the indicated phase. 
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Results of Test 2 


This appendix contains the results for Cases 1, 3, 7, 
and 18. The planned test times were generated for a pro-~ 
bability of survival of .99, 
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APPENDIX C 
Results of Test MOD1 


l. Test MOD1 

Test MOD1 used the AMSAA model to estimate the failure 
rate of the items tested except as described below. A more 
detailed discussion of the molification can be found in 
Section V-D. 

Use of the AMSAA model was modified as follows: 

a. The point estimate of the failure rate, 


number of failures during a phase 


re 


total test time during a phase 


was used whenever cumulative test time over all items was 
less than 10 hours. 

b. The slope of the reliability growth pattern was 
estimated using the current point estimator, Pp? and the 


previous estimate, Pr? of the failure rate as follows: 


Increasing slope was then determined from the following 


relationship: 


If the slope was increasing, the point estimate, r_, was 
used as the current estimate of the failure rate, Then, 
the AMSAA model was reinitialized, that is, time and failures 
prior to an increasing slope were not considered in future 
estimates made by the model. 
2. The Results 

Results for Cases 4, 6, 15, and 18 are presented here as 
representative of the effect the modifications had on the 
performance of the model. A detailed description of the 


format of the results can be found in Appendix A. 
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APPENDIX D 


Results of Test MOD2 


l. Test MOD2 

Test MOD2 used the AMSAA todel to estimate the failure 
rate of the items tested except as described below. A more 
detailed discussion of the modification can be found in 
Section V-D. 

Use of the AMSAA model was modified as follows: 


a. The point estimate of the failure rate, 


number of failures during a phase 


total test time during a phase 


was used whenever cumulative test time over all items was 
less than 10 hours. 

b. The slope of the reliability growth pattern was 

| estimated using the current point estimate, ry? and the 


: Aa 
ry 


t previous estimate, Pil > of the failure rate as follows: 


Increasing slope was then determined from the following 


relationship; 
t 


| 
} 
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If the slope was increasing, the point estimate, rp? was 
used as the current estimate of the failure rate. Then, 
the AMSAA model was reinitialized, that is, time and failures 
prior to an increasing slope were not considered in future 
estimates made by the model. 
2, The Results 

Results for Cases 4, 6, 15, and 18 are presented here as 
representative of the effect the modifications had on the 


performance of the model. A detailed description of the 


format of the results can be found in Appendix A. 
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APPENDIX E 


Results of Test MOD3 


l. Test MOD3 

Test MOD3 used the AMSAA model to estimate the failure 
rate of the items tested except as described below. A more 
detailed discussion of the modification can be found in 
Section V-D. 

Use of the AMSAA model was modified as follows: 


a. The point estimate of the failure rate, 


number of failures during a phase 


P total test time during a phase 


was used whenever cumulative test time over all items was 
less than 10 hours. 

b. The slope of the reliability growth pattern was 
estimated using the current point estimate, tr and the two 


a a 


previous estimates of the failure rate, Pi, andr, 


i-2° The 


slope estimated was in two parts, 


The estimate of the slope was considered increasing if both 
parts of the estimate were determined to be increasing as 


follows: 


130 


and 


ry > 


If the slope was increasing, the point estimate, 5? was 
used as the current estimate of the failure rate. Then, 
the AMSAA model was reinitialized, that is, time and failures 
prior to an increasing slope were not considered in future 
estimates made by the model. 
2. The Results 

Results for Cases 4, 6, 15, and 18 are presented here as 
representative of the effect the modifications had on the 


performance of the model. A detailed description of the 


format of the results can be found in Appendix A. 
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APPENDIX F 


Results of Test MOD4 


1. Test MOD4 

Test MOD4 used the AMSAA model to estimate the failure 
rate of the items tested except as described below. A more 
detailed discussion of the modification can be found in 
Section V-D. 

Use of the AMSAA model was modified as follows: 


a. The point estimate of the failure rate, 


number of failures during a phase 


P total test time during a phase 


was used whenever cumulative test time over all items was 
less than 10 hours. 

b. The slope of the reliability growth pattern was 
estimated using the current point estimate, ry? and the 


previous estimates of the failure rate, Pid and Psp: The 


slope estimated was in two parts, 


The estimate of the slope was considered increasing if both 
parts of the estimate were determined to be increasing as 


follows; 
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and 


If the slope was increasing, the point estimate, r_, was 
used as the current estimate of the failure rate. Then, the 
AMSAA moclel was reinitialized, that is, time and failures 
prior to an increasing slope were not considered in future 
estimates made by the model. 


2. The Results 


Results for Cases 4, 6, 15, and 18 are presented here as 
representative of the effect the modifications had on the 
performance of the model. A detailed description of the 


format of the results can be found in Appendix A. 
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